Terminology

??? Database
What do we call this database? We have been loosely referring to it as the metrics database, but I do not know if that is broad enough. We already know we are going to be storing events in this database. And even within metrics nomenclature, there seems to be a tendency to call out call time data separately from traits and numeric metrics.

Cassandra Node
A running instance of Cassandra. A cassandra node is typically started with the cassandra script on Linux and Unix platforms or cassandra.bat batch file on Windows. The main entry point for a Cassandra node is the class org.apache.cassandra.thrift.CassandraDaemon.

Embedded Node
A Cassandra node that runs in and shares the same JVM process with other applications. Embedding a Cassandra node in the RHQ server for example, would mean that the node is running in the same JVM as the RHQ server.

Embedded Cluster
Every node in the cluster runs on the same machine, and each node binds to a localhost alias, e.g., 127.0.0.1, 127.0.0.2, etc. More specifically, the localhost alias will be used for both inter-node communication (gossip) and for client, thrift RPC requests. Nodes in an embedded cluster can only receive requests from the same machine. It cannot talk to other nodes or clients running on other machines.

An embedded cluster will be used by default during development. An embedded cluster will be used along side the dev-container. Another, separate embedded cluster will be used for automated tests. An embedded cluster might also be useful for demos and even for small installations.

Non-embedded Cluster
Every node in the cluster binds to a hostname or IP address that is visible on the network. More specifically, the node can receive gossip (inter-node communication) requests as well as client, thrift RPC requests from other machines on the network. Note that it is possible in a non-embedded cluster to run multiple nodes on the same machine. It requires having a dedicated network interface for each node, and that interface needs to be visible on the network.

Dev Cluster
An embedded cluster used in conjunction with the rhqdev database and dev container.

Test Cluster
An embedded cluster used for automated tests. It is used in conjunction with the rhq database which is also used for automated tests.

Scaling Out
Adding more nodes to a cluster.

Scaling In
Removing nodes from a cluster.

Unmanaged Deployment

Installing, configuring, and starting an embedded cluster or node that is part of a non-embedded cluster without going through the agent. The same Ant-based bundles code is used as with any other bundle deployment. It is just that the bundle deployment is not executed through the agent.

Managed Deployment

Installing, configuring, and starting an embedded cluster, an embedded cluster node, a non-embedded cluster, or a non-embedded cluster node. This type of deployment operation is carried out by the agent.

Cassandra Configuration Changes

We do not rely on an external Cassandra installation. Instead we embed and deploy our own Cassandra installation similar to what we do with the JBoss AS in which RHQ runs. This allows us to more easily track and manage configuration changes. The following table briefly summarizes the configuration files that we modify and manage during the deployment process.

<rhq:input-property name="rhq.cassandra.rpc_port"
                    description="The port Cassandra listens on for Thrift requests"
                    required="false"
                    defaultValue="9160"
                    type="integer"/>

rhq.cassandra.rpc-port=${rhq.cassandra.rpc-port}

private Integer rpcPort;
.
.
.
public int getRpcPort() {
    return rpcPort;
}

public void setRpcPort(int rpcPort) {
    if (rpcPort == null) {
        this.rpcPort = rpcPort;
    }
}

Server Start Up Use Cases

These use cases focus on deployment of RHQ's Cassandra cluster at server start up. For now you can assume that there are no schema changes with each scenario. The scenarios are intended to include development and testing environments, small production deployments, and large production deployments.

U1 Start RHQ server using embedded cluster where cluster is not yet installed.

1. Start RHQ server in maintenance mode

2. Store Cassandra bundle and bundle version in database.

3. Install embedded cluster using embedded cluster manager.

4. Start cluster using embedded cluster manager.

5. Install RHQ schema.

6. Kick off background job that runs periodically and attempts to connect to cluster.

   1. Once a conection is established to a node, move the server from maintenance mode to normal mode.

7. Start agent.

8. Automatically import nodes into inventory.

9. Create resource group for cluster. (should this by a dynagoup?)

UI does not specify whether or the agent is running because in either case do follow the same steps. If the agent is running, we could install and start the cluster by submitting a bundle deployment. Since this is an embedded cluster, we rely on an embedded deployment regardless of whether or not the agent is running. This helps simplify the start up logic.

U2 Start RHQ server using embedded cluster where cluster is installed, cluster is not running.

1. Start RHQ server in maintenance mode.

2. Start cluster using embedded cluster manager.

3. Kick off background job that runs periodically and attempts to connect to cluster.

   1. Once a connection is established to a node, move the server from maintenance mode to normal mode.

4. Start agent.

The fact that the cluster is already installed necessarily implies that the Cassandra bundle already exists in the database and that the schema already exists. Regardless of whether or not the agent is running, we rely on the embedded cluster manager to start the cluster. This helps simplify the start up logic.

U3 Start RHQ server using embedded cluster where cluster is installed, cluster is running.

1. Start RHQ server in maintenance mode.

2. Kick off background job that runs periodically and attempts to connect to cluster.

   1. Once a connection is established to a node, move the server from maintenance mode to normal mode.

3. Start agent.

U4 Start RHQ server using non-embedded cluster where cluster is not yet installed. Agent is not yet installed

1. Run db install script.

   1. Installs Cassandra node using embedded deployment.

   2. Start Cassandra node.

   3. Install Schema on node.

   4. Install agent.

2. Start RHQ server in maintenance mode.

3. Store Cassandra bundle and bundle version in database.

4. Kick off background job that runs periodically and attempts to connect to cluster.

   1. Once a conection is established to a node, move the server from maintenance mode to normal mode.

5. Start agent.

6. Automatically import nodes into inventory.

7. Create resource group for cluster.

We want the node up and running as quickly as possible so that the server can establish a connection to the cluster. Using an embedded deployment helps accomplish this. The db install script also installs the agent to ensure that the requirement of having an agent running along side the Cassandra node is satisfied.

U5 Start RHQ server using non-embedded cluster where cluster is installed, cluster is not running. Cluster nodes are in inventory.

1. Start RHQ server in maintenance mode.

2. Start agents on cluster node machines.

3. Start cluster using non-embedded cluster manager.

4. Kick off background job that runs periodically and attempts to connect to cluster.

   1. Once a connection is established to a node, move the server from maintenance mode to normal mode.

In this scenario the cluster is already installed but not running. We can assume that the cluster is in inventory; otherwise, we are talking about use case U4. The cluster is started using the non-embedded cluster manager. This in effect means that the RHQ server sends request to agents to start the cluster nodes.

U6 Start RHQ server using non-embedded cluster where cluster is installed, cluster is running.

1. Start RHQ server in maintenance mode.

2. Kick off background job that runs periodically and attempts to connect to cluster.

   1. Once a connection is established to a node, move the server from maintenance mode to normal mode.

3. Start agents.

Server Installation Use Cases

These use cases involve both new installs as well as upgrades. The focus is around the deployment of the metrics database cluster. For the first iteration of this, we plan on making the following changes for the installer and for the new database installer:

• Update the installer so that the user can enter the hostname of machines on which database nodes will run.

  • The user will be required to specify at least one hostname.

  • The user can specify more than one hostname.

• The user will install a database node similar to how he installs the agent.

  • Download the database installer.

  • Run the installer, e.g., java -jar rhq-metrics-db.jar --install /path/to/install/dir

U1 User performs new installation with single Cassandra node on the same machine on which the RHQ server is running.

1. Start installer.

2. For metrics database, specify the hostname of the machine on which the installer is running

3. Finish the installer wizard.

4. Log into RHQ.

5. Verify that the server is in maintenance mode.

   1. Verify that in the server log you see the warning message, "<RHQ_SERVER> is unable to connect to any Cassandra node. Server will go into maintenance mode."

6. Download the database installer.

7. Run database installer

   1. TODO Provide info on installer options. User may specify options like install directory, data directory, log directory, commit log directory, etc.

8. Start database node.

9. Wait a few minutes and verify that the RHQ server operating mode has changed from maintenance to normal.

10. Install and start agent.

    1. Wait for agent to run discovery and send initial inventory report.

11. Verify that Cassandra node is automatically imported into inventory.

12. Verify that RHQ Cassandra Cluster resource group exists and has the node as a member.

U2 User performs new installation with two node cluster.

1. Start installer.

2. Specify hostnames of machines on which database nodes will run.

3. Finish the installer wizard.

4. Log into RHQ.

5. Verify that the server is in maintenance mode.

   1. Verify that in the server log you see the warning message, "<RHQ_SERVER> is unable to connect to any Cassandra node. Server will go into maintenance mode."

6. Download the database installer to each machine on which database nodes will run.

7. Run database installer on each machine.

   1. TODO Provide info on installer options. User may specify options like install directory, data directory, log directory, commit log directory, etc.

8. Start database nodes.

9. Wait a few minutes and verify that the RHQ server operating mode has changed from maintenance to normal.

10. Install and start agent on each machine.

    1. Wait for agents to run discovery and send initial inventory reports.

11. Verify that Cassandra nodes are automatically imported into inventory.

12. Verify that RHQ Cassandra Cluster resource group exists and has both nodes as members.

Provisioning

This section is a follow up on the server starter use cases to further discuss how, when, and where provisioning (via bundles) is and will be used.

Using CQL

The initial back end prototype was developed using the Cassandra client library Hector, which provides among other things an abstraction layer over Cassandra's Thrift API. When I was developing the prototype, I consider using a CQL script for defining the schema. I ultimately decided against it because at the time for the following reasons:

• CQL was undergoing frequent changes making it more difficult to learn.

• CQL and the cqlsh tool were not as well documented as the Cassandra CLI.

• CQL and cqlsh seemed less stable than the CLI in large part due to the frequent changes.

For these reasons I chose to create the schema using a CLI script. When we decided to move forward with Cassandra I expressed that using CQL for client access in application code was not a viable option because of the inability to create dynamic column families/wide rows and the inability to perform range queries against those wide rows. It turns out that you can create dynamic column families/wide rows and query them using CQL. The key is that the table has to use a composite primary key, and the table must include at least one column that is not part of the primary key. 

It is important to understand the differences between the CLI and CQL versions of the schema. Let's look at the raw_metrics column family to point out those differences.

create column family raw_metrics
    with comparator = DateType and
    default_validation_class = DoubleType and
    key_validation_class = Int32Type;

Note that no columns are declared. All columns are dynamic. Here is how the column family looks with some data.

The row key is the schedule ID. Column names are the timestamps of the metric data. Column values are the actual values of the metric data. Now let's look at the CQL version.

CREATE TABLE raw_metrics (
    schedule_id int,
    time timestamp,
    value double,
    PRIMARY KEY (schedule_id, time)
);

This is pretty straightforward and looks fairly similar to what we have with the CLI version. The next diagram highlights some important differences.

The row key is still the schedule ID. The column name is now a composite that consists of two components. In addition to the timestamp, the string literal value is stored as part of the column name. There was some concern about the additional disk space requirements, but the following section on storage requirements shows that there minimal overhead as a result of SSTables being compressed.

Using CQL for client access provides us with several advantages. We will be better able to leverage existing tooling. We get some ad hoc query capability. Developers get to use familiar SQL syntax instead of having to learn another library. Lastly, we will have to write less code for read and write operations. This all lead to easier maintenance for the long term. I wrote up a blog post that offers a more detailed explanation why the move to CQL makes sense. You can find it here.

Storage Requirements

One of the questions that has come is how much disk space is required to store a week, a month, a year, etc. of raw metric data. I have done some preliminary analysis to start answering these questions.